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a program for dropouts. New developments in methods include 
poeioeeneatarsy: the mapping of the qualitatively different ways 
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with other fields such as law, journalism, and economics. The growth 
of qualitative approaches is considered as a sign of greater 
ecumenism of methods and a possible paradigm shift in the qualitative 
ela A five-page reference list concludes the document. 

LPG 


RAEAEEKRARERERERREERRERERRERRERRRERREREEREREREEREREREEEKEEEREEEEEEREREKREEEREERE 


* Reproductions supplied by EDRS are the best that can be made * 
x 


from the original document. > 
RAREEERAEARRERRERRAREREREREEREREREREERRERERKREREREREEREREREERERERRREREREREEEEK 


Tin £70 320 


at 
we 
fee) 
=f 
oc 
CQ) 
[ am) 
La 


Qualitative Approaches to Evaluating Education 


“PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 


AM Falter aan _ 


TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC).” 


Presented at the American Educational Research Association 


David M. Fetterman 
School of Education 
Stanford University 
Stanford, CA 94305 


Meeting, Washington, 0.C. April 20-24, 1987. 


BEST COPY AVAILABLE 


U.S. DEPARTMENT OF EDUCATION 
Otfice of Educational Research and improvement 
EDUCATIONAL RESOURCES INFOR' 

CENTER (ERIC) ne 

Sinn document has been reproduced as 
rom so 

wonuier person or organization 
Minor changes have bee: 

reproduction quality eae eee 


© Points of ew or opinions stated in this docu- 
ment do not necessarily represent official 
OER! position or policy 


Annual 


© School of Education, Stanford University, Stanford, CA 94305 


Acknowledgements ei 


I would like to thank my colleagues in this review for helping me § to 
enhance an understanding of and a sensitivity to qualitative approaches 
in educational research. In addition, Deborah S. Waxman's assistance in 
the preparation of this manuscript is greatly appreciated. 


Qualitative educational research is not a monolithic entity. A 
multitude of qualitative approaches exist. They may be scientifically 
based or artistically oriented. One approach may appear radically 
phenomenological, another mildly positivistic in style, tone, and 
formation. Epistemological and methodological pluralism is a reality in 
educational research. This article explores this qualitative diversity 
and, in the process, dispels the myth of a homogeneous enterprise. Some 
of the most common approaches in the field -- including ethnography, 
naturalistic inquiry, connoisseurship and criticism, and a few 
completely mew qualitative approaches -- are briefly discussed in this 
presentation. These approaches are illustrated with the work of their 
founders or major Serre Py 


In some of these reviews, a specific approach is discussed; in 


other portions of this review, the issues that shape and distinguish one 


aporoach from another are examined. Arguments are openly aired and 
hopes for reconciliation are offered. No attempt is made to 
exhaustively review each approach. (For a more detailed review of each 


approach see Fetterman in press b.) 

Qualitative approaches in the field of educational research 
represent a wealth of useful, practical alternatives designed to add to 
the educational researcher's arsenal. Comparing and contrasting these 
approaches clarifies their relationship to one another and ensures a 
more appropriate and accurate appraisal of the individual approaches. 
Critics often confuse one qualitative approach with another. This 
misperception has caused erroneous or misleading evaluations of a given 
approach. Typically, the wrong criteria are used to assess the utility 
of an approach. Criteria to determine the validity of ethnography may 


be used inappropriately to determine the value of connoisseurship = and 


criticism, and the criteria for evaluating naturalistic inquiry are 
often similarly egeetnninae In addition, some researchers have 
haphazardly mixed elements of different qualitative approaches in a 
single study without regard for the fact that each approach has its own 
set of standards, thus jeopardizing the credibility of research 
findings. Elements of different approaches can combine in a_ single 
study if the evaluator is knowledgeable about the various approaches and 
aware of the consequences of mixing and matching. The validity of one 
qualitative approach can be enhanced when supplemented by the techniques 


of another. However, an undisciplined approach to combining qualitative 


approaches can undermine the 12st interesting study. 


This discussions take place in a larger paradigmatic context: that 
of a silent scientific revolution in educational research. The change 
is most visable in educational evaluation. As is the case for many 
fields of scientific endeavor, educational evaluation is experiencing a 
change in direction. A critical component of this change is a shift in 
the paradigms underlying the method and aim of suede. A marked 
shift is taking place in the professional allegiance of evaluators. 
Increasingly, evaluators are turning away from traditional positivist 
approaches and toward the acceptance and use of phenomenological or 
qualitative concepts and techniques in educational evaluation. As with 
any change in science, the shift is gradual, involving both subjective 
and objective considerations. Thomas Kuhn, the preeminent historian and 
philosopher of science, who explored the evolution of scientific 


revolutions, explained that the acceptance of a new paradigm depends on 


the phenomena of prior crisis and faith, as well as numerous hard-headed 


arguments: 


The man who embraces a new paradigm at an early stage must 
often do so in defiance of the evidence provided by problem- 
solving. He must, that is, have faith that the new paradigm 
will succeed with the many large problems that confront it, 
knowing only that the older paradigm has failed with a few. 
A decision of that kind can only be made on faith. 

That is one of the reasons why prior crisis proves so 
important. Scientists who have not experienced it will 
seldom renounce the hard evidence of problem-solving to 
follow what may easily prove and will be widely regarded as a 
will-o'-the-wisp. But crisis alone is not enough. There 
must also be a basis, though it need be neither rational nor 
ultimately correct, for faith in the particular candidate 
chosen.... 

This is not to suggest that new paradigms triumph 
ultimately through some mystical aesthetic. On the contrary, 
very few men desert a tradition for these reasons alone. 
Often those who do, turn out to have been misled. But if a 
paradigm is ever to triumph it must gain some first 
supporters, men who will develop it to the point where 
hard-headed arguments can be produced and multiplied. find 
even these arguments, when they come, are not individually 
decisive. Because scientists are reasonable men, one or 
another argument will ultimately persuade many of them. But 
there is no single argument that can or should persuade them 
all. Rather than a single group conversion, what occurs is 


an increasing shift im the distribution of professional 


allegiances (1962, p.158). 


The conversion experience that Kuhn speaks of does not occur avanntahts, 
It is not unusual to observe "lifelong resistance particularly from 
those whose productive careers have committed them to an older tradition 
of normal science..." (p.151). Donald Campbell (1974) and Lee Cronbach 
(197S) stand as rare exceptions to this pattern. Prominent proponents 
of the dominant (positivistic) paradigm, they have both taken firm 
positions in favor of the use of qualitative methods. Im fact, Campbell 
(1979) has stated that 

where such (qualitative) svallieticns are contrary to the 

quantitative results,» the quantitative results should be 

regarded as suspect until the reasons for the discrepancy are 

well understood (p. 53). 
Educational researchers who continue to display resistance and 
uncertainty are usually unfamiliar with qualitative approaches. This 
discussion addresses this problem by presenting a set of standard 
qualitative approaches that have emerged in the course of this silent 
scientific revolution. 

Revolutionary change occurs in many stages from innovation to 
acceptance. Typically, only a few innovations reach the acceptance 
stage. The qualitative classics in this review represent accepted 
innovations in the evaluation enterprise. Acceptance creates a 
hospitable environment for future innovations. Novel approaches reach 
the surface of awareness in this kind of environment -- approaches’ that 
under less accepting and flexible circumstances would never see _ the 
light of day. New developments, either end up in a suitably obscure 


place in the archives or reach the light and in turn light the way to 


the future. Reaching the acceptance stage generally means adapting an 


innovation to the mainstream --‘to make it more familar to potential 
adopters. This process can be accomplished by modifying superficial or 
substantive elements of the paradigm to make it more palatable to the 
dominant group. During this adaptation period the brainstorming phase 
comes to a close, and it is time to regroup. Successful change agents 
are able to identify the salient elements of an innovation from the 
potential adopters’ perspective and to promote or proselytize, focusing 
on the significant features of the innovation. Similarly, perceived 
weaknesses that threaten the validity or credibility of the innovation 


must be addressed if the innovation is to be fully assimilated into the 


superordinate group. The bottom line, however, in any marketing 
strategy is the product. Without a finished product, all the 
advertising or proselytizing in the world is meaningless. In 


evaluation, reports represent one of the most convincing arguments for 
qualitative approaches. They can stand the test of time and they can be 
evaluated on their own terms, They are either convincing or 
unconvincing, useful or useless. At this stage of an innovation, the 
idea comes to fruition -- for better or worse. 

A fundamental element of the acceptance process is communication. 
The continuing qualitative-quantitative debate is an important part of 
this process (see Smith and Heshusius 1986; also see Phillips 1983 and 
Soltis 1984). One need only scratch the surface of the qualitative- 
quantitative debate to understand that the terms quantitative and 
qualitative are in themselves misleading. The terms are commonly 
accepted handles for both the contrasting paradigms and the methods 
associated with them. However, each paradigm employs both quantitative 


and qualitative methods. Certainly, adherents of the dominant 


are more likely to use experimental and quasi- 


quantitative paradigm 
“experimental tools, while qualitative researchers are more likely to 
employ more descriptive techniques. Focusing on methods, however, is 
like focusing on the symptoms rather than the cause of a disease. 
Methods are manifestations of a manifold religion we call science. The 
fundamental differences between scholarly orders is based-= on 
philosophical and epistomological, not methodological, grounds. The 
contrast in this case revolves around the philosophical positions of 
positivism and phenomenology. Typically, positivists search for social 
facts apart from the subjective perceptions of individuals. In 
contrast, phenomenologicably oriented researchers seek to understand 
human behavior from the “insider's” perspective. Their most significant 
reality or set of realities is found in the subjective realities of 
human perception. Essentially, a phenomenologically oriented researcher 
argues that what people believe to be true is more important than any 
objective reality: people act on what they believe. Moreover, there are 
real consequences to their actions. This basic philosophical 
difference, in conjunction with the social and psychological attributes 
of the individual researcher, sets the tone for research. These 
characteristics shape the research endeavor, from the methods used to 
the types of questions asked. These pedagogical distinctions become 
somewhat muddled in practice, however, because a continuum runs” from 
reform through orthodox adherence to a paradigm. Moreover, as the 
research evolves, the researcher may alter his or her vision. The work 
of such anthropologists as Russel Bernard and Marvin Harris is designed 
and conducted froma phenomenologically oriented perspective. Both 
attempt to extrapolate from their data 


anthropologists, however, 


external Durkheimian social facts in a classical positivistic tradition. 
Similarly, most qualitative researchers attempt to communicate their 
insights and research findings to positivists -- the dominant culture in 
educational research -- in the language of their host aati.” 
However, communication between contrasting cultures often produces 
conflict and debate. In our disputes, we forget that we are one family 
in pursuit of knowledge. The current dispute at times echoes’ the 
tensions that existed in the !6th century between believers in the 
Copernican theory of the universe and the Ptolemaic established order, 
which preached that the earth was the center of the galaxy. Copernicus’ 
theory was anathema to the church and a threat to the established way of 
thinking about the world aus ourselves. Skeptical thinkers, including 
Galileo and Kepler, produced treatises that helped bute a case for an 
alternative way of viewing the solar bougsk It was a gradual shift in 
professional allegiances, in practice not that much different from the 
current shift in allegiance in educational research. No promises can be 
made for the powers of a new paradigm. All that can be said is that a 
Qualitative paradigm offers a new set of explanations of our educational 
system. It also enables researchers to ask new questions, answer 
different kinds of questions, and re-address old ones. In essence, it 
has worked in a number of areas where the dominant paradigm has failed 
or is simply inappropriate. 

This shift in allegiance is not a simple linear development. 
Qualitative evaluation has manifested itself in a variety of forms, and 
entirely new paradigmatic transformations have occurred. Some new 
approaches are the result of a Hegelian synthesis of paradigms, others 


-- such as phenomenography -- appear to have emerged more independently. 


Some of the most effective of these approaches have been selected for 
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this review to document the development of this gradual shift in 
professional allegiances among educational researchers. The reasons for 
this realignment vary. Many individuals have been convinced of the 
utility of this new paradigm “through some mystical aesthetic." 
Increasingly, however, individuals seriously dissatisfied with the 
results of their old tools are making the case for other qualitative 
approaches in education "to a point where hard headed arguments can be 
produced and multiplied." | 

One of the first formal collections dealing with the paradigmatic 
debate was presented by Cook and Reichardt (1979) in Qualitative and 


5 
Quantitative Methods in Evaluation Research. This collection presented 


an excellent introduction to the issues surrounding these contrasting 
paradigms and discussed some of the strengths and weaknesses of each 
approach. In addition, the authors have provided a forum for the debate 
and have established some boundaries on the discussion. They have, 
however, been criticized for "making explicit many of the 
misunderstandings that have emerged as a result of writings on 
alternative paradigms" (Patton 1980:228 . The real problem critics have 
with their collection may be a result not of scholarly misunderstanding 
but of the collection’s positivist perspectives on a phenomenological 
endeavor, ; 

The first phenomenologically oriented presentation of this debate 
was presented by Fetterman (1984) in Ethnography in Educational 
Evaluation. The contributors were cultural brokers, agents of change 
attempting to diffuse a paradigm. They demonstrated the utility and 


centrality of ethnography in educational evaluation. Ethnography in 


Educational Evaluation presented a continuum of practices within one 
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qualitatively oriented tradition -- ethnography. It captured what doing 


ethnographic educational evaluation means from the emic or “insider's” 
perspective. 

Educational Evaluation: Ethnography in Theory, Practice, and 
Politics (Fetterman and Pitman 1986) was designed to build on the 
foundation laid by the previous work. It presents the latest 
developments in the emerging field of ethnographic educational 
evaluation from an anthropological perspective. This book demonstrated 
various degrees of assimilation, acculturation, and deacculturation to 
the dominant context of evaluation. The emphasis on the explicit use of 
anthropological theory calls for a return to the basic elements of 
native anthropological wtteipae The practice and politics chapters 
demonstrate how to integrate qualitative and quantitative data within a 
single study. Consequently, some chapters sound more sociological than 
others, some more positivistic thar phenomenological. The aim, however, 
was to present a continuum of what is happening in practice during this 
stage of the cultural exchange. 

This discussion attempts to paint on the same canvas with much 
broader strokes -- painting a portrait of paradigmatic change. Instead 
of presenting a continuum of practices in monochrome, this sketch 
continues the debate by presenting an insight into the rainbow of 
colorful issues and approaches within a qualitative dimension. The 
approaches selected for presentation stand as useful alternatives to the 
dominant paradigm. Moreover, this collection is presented because, as 
Kuhn explains, "no single argument...can or should persuade them all." 

Structurally, this review 1s divided into four sections: 
qualitative eae.” new developments, regrouping, and conclusion. 


The heart of this review lies in the qualitative classics section. Here 
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the dominant qualitative forms of evaluation are displayed, including 
ethnography, naturalistic inquiry, wenerie pragmatic (sociological ) 
qualitative inquiry, and connoisseurship/criticism. Approaches are 
distinquished from one another, basic issues are addressed, and 
unresolved disputes are discussed. 

A natural tendency of any radical change is the emergence of 
splinter groups, new factions, and entirely new developments. This 
section presents a few of these marginal but potentially significant 
evolutionary changes -- metaphors and phenomenography. The need to 
regroup demonstrates a natural tendency in any process of change -- to 
assess where we are and identify the next steps necessary to routinize 
the movement within the context of mainstream traditions. Finally, the 
formal conclusion simply provides a broader perspective to this exchange 
by presenting a mild criticism of each approach. 

Qualitative Classics 

Although a young science : 1n comparison to the physical sciences, 
anthropology has the distinction of being one of the oldest qualitative 
traditions in academia. Anthropology has a multitude of built-in 
quality controls, with an emphasis on ensuring validity, amd thus has 
become one of the most widely accepted qualitative approaches among 
positivists. Ethnography, a subskill in anthropology, has become firmly 
rooted in educational evaluation. In “Ethnography in Educational 
Research: The Dynamics of Diffusion" (Fetterman 1982) and "Ethnographic 
Educational Evaluation,” (Fetterman in press a), Fetterman discusses the 
origins of this field, key elements of this approach including 
techniques and a cultural interpretation, and required adaptations. He 


focuses on a national ethnographic evaluation of a program for dropouts 
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that helped legitimize this source of inquiry in evaluation circles. 


The study was multilevel and multidimensional in nature. The 
evaluation examined classrooms, administrative structures, community 
environments, local and national program affiliates, and governmental 
agencies. In addition, the roles of federal involvement, evaluation 
design, and the rola of reinforcing world views were examined. This 
exploration also contributed to the study of cultural transmission, 
focusing on such mechanisms as the role of program ethos, rites of 
solidarity, and rites of passage. In addition, it demonstrated the 
significance of contextualizing data on program, evaluation, and federal 
levels. An attempt was made to demythologize the qualitative- 
quantitative dichotomy in research. Ethnography requires a good mixture 
of qualitative and quantitative data to discern attitudinal changes and 
to understand typical quantitative criteria such as attendence, 
turnover, graduation, and placement figures. Moreover, this study 
demonstrated how integrating qualitative and quantitative data is 
possible. Finally, it suggested the policy relevance of the qualitative 
approach. Sensitivity to policy language and governmental timelines and 
a demonstrated ability to make significant programmatic and policy 
recommendations has helped ethnographic educational evaluation find 
fertile ground. 

Intracultural diversity is characteristic of an evolutionc , 
development in any. discipline. Philosophical and methodological 
arguments abound within the qualitative community. These arguments help 
refine the direction of the field. Miles and Huberman argue about 
methods and canons for analysis required to translate qualitative 


findings to mainstream educational researchers in a credible fashion. 
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Guba and Lincoln argue about the theoretical and epistemological issues 


and, like Wolcott in educational anthropology, play the spoiler role to 
maintain the integrity of their approach. They represent a conservative 
force, preventing excessive adaptation and modification. In 
Naturalistic Inquiry ‘Lincoln and Guba 1985) and in "Do Inquiry 
Paradigms Imply Inquiry Methodologies?," (Guba and Lincoln in press), 


they contrast the scientific, positivistic paradigm with a naturalistic 


methodology. According to the authors, the alternative paradigm 
“represents a rival ontological, epistemological, and axiological 
posture” for adherents of the conventional paradigm. They argue with 


positions held by Miles and Huberman, Cook and Reichardt, and Patton who 
attempt an ecumenical leddine of methods and/or aie shifting of 
paradigms. Guba and Lincoln suggest that these positions confuse 
methodology (paradigms) with methods (tools and techniques). They 
explain that nothing is intrinsically naturalistic or positivistic about 
methods. The classification of an approach depends on the researchers’ 
intent or purpose and how they use their tools. Moreover, the authors 
argue that elements of the conventional and the alternative paradigms 
cannot be mixed without resulting in complete ruin. Guba and Lincoln 
present the axiomatic differences between positivistic and naturalistic 
paradigms, the differences in contexts of discovery and verification, 
and the negotiated or collaborative nature of naturalistic inquiry in 
comparison to exclusively exogenous or endogenous (locus of inquiry) 
approaches. They also briefly contrast the linear, rational, and closed 
methodology of the conventional paradigm with the circular, interactive, 
hermeneutic, and intuitive character of the naturalistic paradigm. Guba 
and Lincoln comment on bounding and the trustworthiness of naturalistic 


inquiry in their work, focusing on internal and external validity, 


reliability and objectivity or credibility, transferability, and 
‘dependability and confirmability. Guba and Lincoln maintain a strong 


position concerning the non-miscibility of the methodologies in any 


proportion." 

Patton presents “a paradigm of choices," in Qualitative Evaluation 
Methods (Patton 1980) and in "Paradigms and Pragmatism" (Patton 1980). 
He agrees with Guba and Lincoln that paradigm distinctions are real and 
useful. However, in marked contrast to their stand, he argues that “one 
can usefully mix methods" without uniformly adhering to a specific 
paradigmatic party line. Patton reviews the paradigm debate, exploring 
assumptions about the connection between paradigms and methods. He 
briefly presents his early lament about the dominance of the 
positivistic over the phenomenological paradigm, Reichardt and Cook’s 
(1979) attack on the untenable conceptualization of two mutually 
exclusive approaches, and Guba and Lincoln’s advocacy of naturalistic 
inquiry over the conventional positivistic paradigm. 

He proceeds to clarify the difference between competing and 
incompatible paradigms, explaining that paradigms do compete for 
resources, but they are not necessarily incompatible in a single study. 
Patton also takes a step beyond logical dichotomies erected to 
distinguish the two paradigms. He presents a revised version of 
Reichardt and Cook’s logical but oversimplified paradigmatic contrast. 
For Patton, the link between methods and paradigms is one of habit and 
training, which place blinders on evaluation practice. 

Patton recognizes the logic behind Guba and Lincoln's position that 
the paradigms are incompatible but argues that pragmatism can overcome 


logical contradictions. Patton advocates the use of "mind shifts bact- 
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and-forth between paradigms within a single evaluation setting" (p. 13). 


Moreover, he has found that if a commitment to an empirical perspective 
exists -- basic pragmatism anda sensitivity to client needs -- the 
other differences can be negotiated. 

Patton recognizes his call for flexibility is an ideal frought with 
difficulties. A multitude of method and measurement choices exist in any 
study. Paradigmatic contrasts are useful pedagogical devices to 
highlight the different values of each approach. But in practice, 
methods choices are made along a_ continuum. Obtrusiveness and 
manipulation may be considered taboo in qualitative approaches, but they 
do exist. The issue is one of intent and degree. 

Patton reflects on ern tendencies in evaluation practice. 
Significant proponents of . the experimental design have endorsed 
qualitative methods and apparently have less resistance to the 
phenomenological paradigm. However, quantitative approaches are still 
dominant. Merging qualitative and quantitative approaches has been 
problematic, but Patton notes that many efforts have been successful. 
In essence, Patton views the debate from a pragmatic empirical 
perspective, viewing what researchers do in practice in comparison to a 
strictly logical or theoretical perspective. Fundamentally, Patton 
attempts to lift the blinders of methodological habit from evaluators 
and to increase the options available to them. 

Eisner presents the role of educational connoisseurship and 
criticism in educational evaluation in, “Educational Connoisseurship and 
Criticism: Their Form and Functions in Educational Evaluation” (Eisner 
1976) and in his article "On the Differences Between Scientific and 
Artistic Approaches to Qualitative Research" (Eisner 1981). 


Connoisseurship and criticism together represent an important 
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alternative in educational research. This option is distinct from other 
qualitative approaches in being epistemologically rooted in the arts 
rather than in Cian, Eisner recommends this alternative to change 
conventional positivistic forms of evaluation. He rejects the concept 
that classroom life is controlled by behavioral laws. Instead, he 
believes evaluation should seek to improve the individual artistry 
demonstrated by individual teachers in unique classroom settings. 
Eisner explains that "“connoisseurship is the art of appreciation, [and] 


criticism is the art of disclosure” (p. 141). Connoisseurship requires 


an awareness and an understanding of the phenomena observed and/or 


experienced. Educational criticism involves description, 
interpretation, and evaluation. Description is thick and detailed, 
capturing the subtleties and the spirit of the moment. Interpretation 


is informed by “social sciences and the practical wisdom born of 
experience in schools" (p. 145). Evaluation requires a value judgment 
about the educational significance of the observation or research 
finding. Eisner discusses two procedures to determine the validity of 
this approach: structural corroboratic., and referential adequacy. 
Structural corroboration refers to the extent to which pieces of the 
puzzle fit together and validate each other. It is similar to the 
process of determining whether the threads of a murder mystery are woven 
into a recognizable (or credible) pattern. Referential adequacy 
involves comparing the critical disclosure with the phenomenon. It 
represents a form of interjudge or'intersubjective agreement. Eisner 
uses art education to illustrate the utility of educational 
connoisseurship and criticism in his discussion. However , the 


application of this approach goes beyond any single discipline. The 
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product of this venture is the reeducation of perception for the 


teacher, the student, the administrator, and the scholar. 
New Developments 

Social conditions must be ripe for change. Smith's exploration in 
alternative research and evaluation methods is rooted in the same social 
order that gave rise to the interest in qualitative methods in this 
review. An increased interest in qualitative approaches together with a 
disillusionment with traditional experimental and quasi-experimental 
approaches facilitated the development of new qualitative methods - 
metaphors. Smith reports the findings of an exploratory National 
Institute of Education project that used other fields as metaphors for 
educational research and Suabunias in, "Mining Metaphors for Methods of 
Practice” (Smith in press). Smith defines a metaphor as a device to use 
"one object to create a new perspective on another" (p.4). In essence, 
the project attempted to view educational research from the perspectives 
of a number of other fields. In addition, metaphors provided an insight 
into alternative techniques, new conceptual distinctions, and possible 
professional roles that might improve educational research and 
evaluation. 

Smith reviews nine metaphors to illustrate the range of methods 
investigated in the study. They include law, journalism, management 
consulting, economics, operations research, geography, photography, and 
music and art. Law as a metaphor offers education such valuable tools 
as legislative histories, the appeals process, and case histories. Law 
also provides the concept of levels of confidence and adversary 
hearings, which can be applied to various evaluation settings. 

A few of the concepts guiding the field of journalism include 


minimum/maximum projections, aborting, and fairness. Journalism also 
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uses such interviewing strategies as circling, filling, and shuffling, 
as well as the key interview. Tracking is another invaluable tool for 
the investigative reporter. The management consultant's role as client- 
oriented diagnostician provides a.model for user-focused evaluators. 
The most significant contribution identified from the field of economics 
included cost-feasibility, cost-utility, cost-benefit, and cost- 
effectiveness analyses. Operations research provided the fewest 
generalizeable methods, in large part aresult of the difficulty of 
translating the tools of this field into a usable form for 
practitioners. However, future applications of mathematical modeling, 
assignment and transportatron methods, decision analysis procedures, and 
operational network displays appear promising. 

Concepts unique to geography, but potentially generalizeable to 
educational research and specifically evaluation, include primitives, 
satisficing, and least protest. In addition, the us”, of maps using 
geocode analysis among other techniques have been useful in analyzing 
the distribution of student achievement data. Mental mapping 1s a 
promising technique to tackle perceptual duta. 

Art, in the forms of photography, music and visual art, and film 
criticism, were difficult to adapt to educational research and 
evaluation, but some results were fruitful, The useful tools adapted 
from photogrephy include sampling techniques, photo-interviewing, and 
theory testing. Techniques drawn from ethnomusicology were similar to 
those used by field workers in social science: preparation for the 
field, rapport, observation, interview, and time management. Watercolor 
painting became a model for educational inquiry, focusing on mastery, 


composing, compelling, and completing concepts. Film criticism tools 
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included thematic matrix analysis, discontinuities in word and image, 
appreciative descriptions, lateral tracking, deep focus cinematography , 
symmetry, and repetition. Smith concludes his discussion with a 
statement about the yield of this exploration and the conditions for 
success in this project. 

Ouring paradigmatic transitions, many alternatives emerge. One of 
the newest developments is phenomenography. Marton presents this new 
qualitative approach in, “Phenomenography: Exploring Different 
Conceptions of Reality” (Marton in press). This approach emerges from 
the qualitative roots of the 1970s, but stands between the alternative 
approaches and the mainstream paradigm. Phenomenography is used to 
study learning and beeing, mapping the qualitatively different ways in 
which people experience or think about various phenomena, such as 
numbers, reacing, and tninking. Marton presents examples of results 
using this approach and discusses the methodological principles 
underlying phenomenography. Phenomenography looks at "the relations 
between human beings and the world around them," focusing on the 


perception itself. For Marton, perception falls between human beings 


and the world around them. Marton recognizes that other established 


traditions have dealt with this domain. However, he is calling for "a 
specialization in its own right" (p7). Categories of description are 
viewed as the outcome of phenomenographic research. Marton discusses 


the concept of replicability for this new qualitative approach, 
separating discovery from identified categories requiring some form of 
intersubjective agreement, He also discusses how phenomenography 
evolved from reflections of mainstream research, measuring and improving 
language proficiency to its present and varied directions. Marton 


refines our understanding of phenomenography by carefully comparing and 
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contrasting it with other qualitative approaches to educational research 


-- specifically phenomenology and siimeueacho Marton concludes with a 
discussion of some of the methodological facets of phenomenography, 
focusing on interviews: educational applications of phenomenography, 
including documenting the effects and noneffects of educational 
treatments; and some implications for an epistemological policy that 
Questions the existing scientific base for teacher education. 
Regrouping 

Firestone and Dawson's “Approaches to Qualitative Data Analysis: 
Intuitive, Procedural, and Intersubjective," (Firestone and Dawson in 
press) marks a transition un the acceptance of the qualitative paradigm. 
They believe qualitative methods have “become an accepted tool in 
educational research." They recognize, however, that their continued 
acceptance and full promise require methodological refinement. They 
explore intuitive, procedural, and intersubjective approaches that aim 
at disciplining “qualitative inquiry without sacrificing subjective 
understanding.” Like each contributor in this review, they are cultural 
brokers. They speak the language of evaluation to convince evaluators 
and other educational researchers that perceived weaknesses in the "new" 
paradigm have an easy remedy. Simultaneously, they are adept code 
switchers, speaking the languages of fieldworker and evaluator in the 
same breath. Their aim is to encourage qualitative researchers to 
refine their own approach while working in the field of evaluation. 

Many stages mark the evolutionary development of a discipline 
(Fetterman 1986). A classic stage involves pulling back and 
regrouping to establish standards commensurate with the mainstream 


rules and regulations of scientific inquiry. An explicit representative 
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of this developmental stejge is presented in, "Drawing Valid Meaning 


from Qualitative Data: Toward a Shared Craft," (Miles and Huberman 


19844). (Also see Qualitative Data Analysis, Miles and Huberman 1984 b.) 


Miles and Huberman argue for an “ecumenical blend of epistemologies 


and procedures.“ However, in general, they leave the epistemological 
debate to others. Instead of focusing on the paradigmatic level, they 
emphasize the practical, methodological level of abstraction. 


Miles and Huberman are concerned that there are “few agreed on 
canons for analysis of qualitative data." They outline a form of data 


analysis and specify methods that provide assurance and credibility to 


the analytical endeavor. The terrain of the quantitative 
researcher’s field is well marked, while the qualitative field 
is “more perilous." For the authors, the "problem is that there 


is an insufficient corpus of reliable, valid, or even minimally agreed- 
on working analysis procedures for qualitative data" (p.2). They 
provide a suggested audit trail from data collection through analysis 
and interpretation. Qualitative data analysis, for Miles and 
Huberman, consists of three components: data reduction, data 


display, and conclusion-drawing and verification. They recommend the 


following methods of improving the data reduction process: 
drawing explicit conceptual frameworks, bounding inquiry (with 
specific research questions), specifying the multitude of sampling 
decisions, and preplanning instrumentation. A variety of interim data 


reduction methods are suggested to prevent “excessive prefocusing and 


bounding," including summary sheets, coding schemes, memos, analysis 
meetings, and interim summaries. 


Miles and Huberman also note that various forms of data display 


improve data analysis, including descriptive and explanatory matrices. 
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Conclusion drawing tactics include counting, noting patterns or themes, 


seeing plausibility, clustering, making metaphors, splitting 
variables, subsuming particulars into the general, factoring, noting 
relations between variables, finding intervening variables, building 
a logical chain of evidence, and making conceptual/theoretical 
coherence. Conclusion verification tactics include checking for 
representativeness, checking for research effects, triangulation, 
weighting the evidence, making contrasts/comparisons, checking the 
meaning of outliers, using extreme cases, ruling out spurious relations, 
replicating a finding, checking out rival explanations, looking for 
negative evidence, and gatting feedback from informants. Miles and 
Huberman conclude with a call for greater sharing of what 
qualitative researchers do when they analyze their data. 

An examination of the strengths and weaknesses of each approach is 
presented below to place the silent scientific revolution in 
paradigmatic perspective. The second half of this discussion consists 
of a mild criticism of the qualitative classics, new developments, and 
the regrouping effort. 

Criticism: Qualitative Classics 

Fetterman presents a case study of a national ethnographic 
evaluation. The study contributed to both basic and policy 
research, It also contributed to quantitative as well as 
qualitatively oriented literature by studying the experimental design 
in the evaluation, as well as the program for dropouts itself. The 
study’s national exposure accounted for as much of its impact on the 
mainstream research community as did the implementation of specific 


qualitative techniques and procedures. This national = study helped 


to demonstrate the value of qualitative approaches to federal 


sponsors and policy decision makers, as well as to campus colleagues. 
However, its strong focus on one national study is also its weakness. A 
‘comparison of many qualitative studies of this scale might have 
improved the salience of the arguments (see Firestone and Herriot 
1984). In addition, a more detailed discussion of the methodology 
of this study, which was specifically tailored to the fiscal and 
timely demands of evaluation research, could profitably have been 
compared with the long-term site visit approach of the Rural Schools 
ethnographic evaluation. In this case, given the benefit of 
hindsight, it appears that gambling on the value of an in-depth look 
at one significant ere and its impact in & number of areas has 
already paid off -- despite the study's necessarily cursory portrayal of 
the ethnographic field. 

Guba and Lincoln are two names” that are synonymous with 
naturalistic inquiry. Their work serves an important purpose -- to 
remind qualitative researchers of basic epistemological roots. Their 
contrast of naturalistic inquiry with positivism is useful and 
important. Researchers who use methods without understanding 
the cosmology of the approach will misuse them. (See Fetterman 1964; 
McCutcheon 1981). Their discussions about credibility, 
transferability, and dependability and confirmability are an attempt 
to bridge a paradigmatic gap in understanding. Their strong stance on 
the non-miscibility of methodologies places them w=: the extreme = end 
of the qualitative family. In espousing this position, the authors 
ignore some real world constraints that shape research. In addition, 
this type of position serves to shut scholars out rather’ than 


accommodating them. However, their position becomes a special 
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demarcation point. As an orthodox sect helps to maintain the purity 
of their religion, the authors help define a continua of nuniieaiase. 
investigators and to maintain the quality of all qualitative endeavors. 

Patton is one of the most pragmatic qualitative researchers in 
the field of qualitative inquiry. His generic qualitative approach is 
sociological in tone. (An anthropological approach is typically guided 
by the culture concept.) His eclectic approach appeals to the 
conventional evaluator. He is able to do the job in a manner that is 
useful to his clients. He stands in direct opposition to Guba and 
Lincoln concerning the mixture of qualitative and quantitative 
methods and paradigms in the same study. Patton might have strengthed 
his argument in this area by emphasizing the importance of training in 
both areas before attempting mind shifts from one approach to another. 
An inadequate understanding of the method and methodology of both 
paradigms could result in disaster if the individual is not properly 
trained for this type of mental desterity. In addition, the 
Achilles’ heel of an overly pragmatic approach is the absence of 
theory to guide practice (see Fetterman 1986; Simon 1986; and Pitman 
and Dobbert 1986). The proof of the pudding, however, is in 
practice, and Patton clearly has been successful in addressing the 
pertinent issues in a timely fashion and ina manner useful to his 
clients. He 15 a major figure in qualitative evaluation. Although 
recognizing that resistance to qualitative forms of inquiry have not 
disappeared, his work has increased the level of acceptance of 
Qualitative inquiry in evaluation. 

Eisner adds another dimension to the qualitative world. His 


work in connoisseurship and criticism 1s based in art, not in science. 
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He offers a refreshing and colorful splash of paint to the canvas. 
The intricate detail of his descriptive portraits resembles the thick 
description of ethnography. They differ only in that ethnography 
is focused by the culture anu and attempts to be nonjudgmentel. 
Connoisseurship and criticism are instead carefully crafted 
judgments of life in the classroom. Such intense appreciation is 
often revealing, opening up a whole new world to the casual observer 
and to the participants themselves. The drawbacks of this approach 
are perceptual. Because connoisseurship and criticism are not based in 
science, those scientists who are the powerbrokers of most educational 
inquiry have difficulty finding the results credible. A more subtle 
problem is that the Sicnaaais 1s conducive to ethnocentric assessments 
and is easily shaped by sociceconomically biased articulate 
perceptions. This problem is compounded when conducting cross-cultural 
research. Moreover, the hidden danger of this approach is that the 
scholar who is immersed in the process of finding a metaphor to 
represent inner reality is tempted to create 1t, producing poetry and 


fiction, not science. This danger is real for all scientists, but 


it is particularly serious for an artist who walks down scientific 


corridors. However , this last concern is anticipated in some 
measure by the use of such procedures as structural corroboration 
and referential siete: These criticisms are not meant to 
detract from the value of the approach, they simply serve as 


Quides to those perusing the gallery of their thoughts. 


Criticism: New Developments 


Smith has found a comfortable home for the metaphor. Like 
Eisner, he values the crystalized and highly focused value of 
metaphors. Unlike E1sner, Smith uses the metaphor to see the problem, 
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whereas Eisner explores the problem in order to see the metaphor that 
° 


‘lies within. Smith's study is a plethora of valuable applications 
and potential applications. He has opened the eyes and ears of many 
evaluators. In some cases, he simply provides useful labels with 


which to descrihe what evaluators are already doing implicitly -- a 
useful contribution in its own right. In addition to testing the 
usefulness of various fields as metaphors for educational research and 
evaluation, he has offered many more concepts to analyze our work. 
Concepts such as levels of confidence and minimum/maximum projections 
and various techniques borrowed from ethnomusicology are clearly 
invaluable. Smith has successfully sensitized evaluators to a 
kaleidoscope of ideas with which to view evaluation problems. One 


caveat 1s that individuals untrained in these areas may apply the 


concepts superficially and inappropriately. However, the thrust of 
Smith's vision 1s to let in more light for our evaluative quest for 
knowledge. 


Marton, like Smith, ea«plores new frontiers. Phenomenography may 
herald @ new era -- or it may be absorbed in mainstream traditions. 
It is always difficult to predict the direction a new field will tate. 
In any case, a contribution has already been made. Marton forces us 
to recognize this form of cognitive research as a definable entity in 
its own right. Phenomenography has existed in various forms as an 
adjunct to a major discipline. This qualitative approach to the 
study of learning and thinking may have tremendous implications 
for the underpinnings of teacher education. A difficulty for 
the advocate of any new discipline or approach is distinguishing it 


from other approaches. In his attempts to define this new 


approach, Marton may have overemphasized the differences and minimized 
the large number of similarities that phenomenography shares with 
phenomenology, ethnography, and sociological field work. This new 
‘approach may appear narrow in scope, but it is still in its infancy; 
only time will determine if this new qualitative approach is fruitful. 
Criticism: Regrouping 

Firestone and Dawson are cultural brokers, trying to 
demonstrate the usefulness of this new approach on the one hand and 
attempting to refine it on the other. The strength of their 
presentation is that they focus on critical issues of concern to 
mainstream evaluators reviewing qualitative approaches. Their 
weakness lies in the subeeunene of their concerns. A multitude of issues 
warrant our attention, imcluding the validity and reliability of 
qualitative approaches. Fundamentally, however, the question is always 
one of tradeoffs. Does one pick depth or breath? Given the 
careful aim they took at the most significant concerns, the authors 
appear to be on target in their choice of depth versus breath. 

Miles and Huberman are two outstanding qualitative 
researchers who have brought methods of analysis to our attention and 
emphasized the importance of various validity and reliability issues 


in qualitative research. They provide a virtual litany of 


techniques to ensure the quality of data collection and analysis. 


Moreover, they, like Firestone and Dawson, are cultural brokers -- 
in this case, explicitly attempting to bring qualitative research 
into the mainstream camp. They have been criticized for attempting 


to impose positivistic standards on a fundamentally phenomenological 
enterprise. Some concern also exists that they may be = structuring 


a qualitative enterprise to death, in spite of caveats to the 


contrary. (See Marshall 1984), A more subtle problem involves the use 


of mainstream terms and concepts to shape and communicate an 
alternative approach -- something is lost in the translation. This 
complaint mirrors Third World scholars’ discussions when they are 
forced to read about their own culture in English with American 


concepts instead of native terms and concepts. These concerns are 


real, but they should not overshadow the significance of these 
two scholars’ accomplishments. No single approach successfully 
communicates the value of qualitative evaluation, and no single manner 
exists to improve it. Thad approach is an important contribution to 


this process. ‘ 


The silent scientific revolution in educational evaluation is like 
a quiet storm. There are no ominous clouds hovering overhead, but the 
power of the storm threatens to tear through the intellectual landscape 
like a tornado. This paradigmatic change is both personal and 
professional in nature. This article views the storm as it travels 
through the rough terrain of qualitative research with a focus on 
evaluation. Mapping the progress cf the storm may help travellors to 
navigate through the clouds to the clearer skies ahead. 

This discussion has many purposes. First, this brief review was 
designed to dispel the notion that qualitative research is a monolithic 
entity: Qualitative approaches are varied and manifold. Second, it 
illustrates the variation in standards. Eacn qualitative approach has 
its own standards and evaluation criteria. This article discusses major 
approaches to facilitate appropriate applications and evaluations cf 


each qualitative approach. A recognition of the intra-cultural 


at 
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diversity within qualitative evaluation will bring about a more 


effective criticism of this art and science. Third, this collection of 
approaches serves as a portable guide to major qualitative approaches 
and arguments in evaluation. (Also see Fetterman in press b for a more 
comprehensive guide. ) Evaluators -- including student evaluators -- 
exposed to a full spectrum of qualitative approaches will be more fully 
equipped to tackle both basic and policy research agendas than will 
those who view the world in terms of one qualitative dimension. 

By openly discussing strengths and weatnesses in the field, this 
review is also designed to help those researchers who are shifting their 
allegiance to a phenomenologically oriented paradigm. This discussion 
may provide some perspective for their own personal struggle with 


loyalty and logic, faith and reason. 
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Footnotes 


1. See Jacobs (1986) for further discussion of this problem. 


2. See Lincoln (1986) for discussion of this paradigmatic shift in 


various disciplines. 


3. Cross-disciplinary communication is also fostered by speaking to 
positivists in native anthropological language when appropriate; 
code switching can also be an effective method of communication (see 


Fetterman 1986). 


4. We would still believe in Ptolemaic cycles and epicycles as 
explanations of the planetary system if not for the persistence of these 
thinkers and the reasonableness of the intellectual community in the 


long run. 


5. Many collections address the general issue of phenomenology and 
logical positivism. In addition, journals -- most noticeably 
Anthropology and Education Quarterly -- have addressed the debate as it 
relates to anthropological and educational research. Cook and 
Reichardt’s (1979) work represents one of the first books to tackle this 


paradigmatic debate directly within the contea~t of evaluation research. 


6. The qualitative classics section represents the most prominent 
standard qualitative approaches used 1n educational evaluation. The 
section is not designed to be an exhaustive list of all qualitative 
approaches or their wajor pranenentas For example, see illuminative 
evaluation (Parlett and Hamilton 1976) for another example of a 
qualitative approach. Also see Hammersley and Atkinson (1983); Goetz 


and LeCompte (1984); and Wolcott (1975, 1984), for additional 


information about ethnography in educational evaluation. Kyle and 


McCutcheon (1984) and Booth (1984) provide insightful illustrations of 


collaborative evaluation. 


7. Ethnography asa formal science is young. However, it has roots 


trace back to the travelogs of Heraclitus. 


8. See Eisner, E., “On the Differences between Scientific and Artistic 


Approaches to Qualitative Research." Educational Researcher, 1981, pp. 


6-9. 


9. This section of the review can be compared with chapter five of my 
forthcoming book The Silent Scientific Revolution: Qualitative 
Approaches to Evaluating Education ‘in press b) to determine how 


academic information can be translated into policy concerns. A critical 
comparison also illustrates wnat type of information can be translated 


and what type of information is omitted. 


12. For a detailed discussion of this topic, see Eisner, E.W. The 
Educational Imagination: Qn the Design and Evaluation of School 


Programs. New York: Publishing Co., 1985, pp. 216-252. 
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